|
DinkumThinkum
|
|
Diviner
|
|
|
|
|
Reged: 09/20/01
|
|
Posts: 4731
|
|
Loc: Luna Penal Colony
|
|
|
A lot of the regulars here know the 'holy litany of load order':
.esm files get loaded first, oldest to newest.
.esp files get loaded second, oldest to newest.
Save Game files get loaded last.
And the critical point for this thread:
Data in files later in the load order overrides data from files loaded earlier.
Unfortunately, I seem to have found an exception to that last bit...
----------------------------------------------------------------------------
This discussion started at the end of the Dirty Saves thread. I'm starting a new thread to avoid confusing that one, and to save people from having to dig through that monster to find this.
The discussion in that thread started with Soar182's
post, reporting that it was possible to trigger doubling just by loading an older mod with new mods loaded.
----------------------------------------------------------------------------
I did some testing, but instead of doubling I wound up with an older mod overriding both a newer mod and a saved game. 
That is not supposed to happen! 
So I took a break and came back and redid the same test, hoping I'd just fouled something up. But the second test had the same results as the earlier one: changes in an older mod were overriding changes in a newer mod and in a saved game.
----------------------------------------------------------------------------
Then I did a new clean install in a new directory and created two new test mods. This time I tested by moving the chest that's near the door of Foryn Gilnith's shack in Seyda Neen.
First, I created two test .esp files:
In Older_ChestByTable.esp, I put the chest by the table.
In Newer_ChestByHammock.esp, I put the chest by the hammock.
I doublechecked on the Data Files screen in the game to be sure Older was listed above Newer, so they were in the right date order.
Then I went into the game to test with those two .esp files:
1. Created a new character with no mods loaded, and saved her just outside the Census Office.
2. Exited the game, then reloaded with the Newer_ChestByHammock.esp loaded. Went to Foryn's shack, and the chest was by the hammock: right where it was supposed to be.
3. Saved the game in Foryn's shack with that mod loaded, and exited the game.
4. Reloaded the game, and enabled both mods this time: Older_ChestByTable.esp and Newer_ChestByHammock.esp.
5. This time, the chest was by the table, which is where it was in Older_ChestByTable.esp, even though both my saved game and the newer .esp should have been overriding and putting the chest by the hammock.
----------------------------------------------------------------------------
In my original test I moved the chest on Arrille's counter around, with the same results (two different test runs): the older mod would override the position of the chest in the newer mod and in the saved game.
To add to my confusion: I did one quick test moving Arrille around, but that didn't duplicate the bug. When I loaded both mods, he was positioned where the newer mod placed him: the older mod didn't override the newer one.
I know that container references are shown as 'temporary' references in cells on the Details list for a mod, while NPCs are just references. So maybe I'll try playing with 'references persist' and see if that has any effect on the results with containers.
----------------------------------------------------------------------------
If anybody can discover that I'm doing something dumb and these results aren't reproducible, I'll be happy to hear it!  I'd rather be wrong on this one than right... 
Right now, though, to me this is yet another reason that clean modding is important. My best guess at the moment is that the bug occurs because both mods are changing the same reference from the original game, and somehow that fouls things up.
--------------------
How to avoid the most common problem encountered by new modders: Dirty Saves: Causes and Cures
Post Extras:
|
|
Klinn
|
|
Diviner
|
|
|
|
|
Reged: 11/20/02
|
|
Posts: 2312
|
|
|
|
|
Some interesting detective work you're doing here, it should really help once we figure out a pattern. Has anyone looked at the contents of a save game file to see if a list of mods-in-use is in there?
I'm speculating that the first time a new mod is added to the mix, no matter if older or newer, the game engine might see that it wasn't on the list of mods in use when the save game was created, and then gives precedence to that mod's info. This theory still allows for the typical dirty save problem, because that happens when you're using a mod with the same name (and would therefore already be on this hypothetical list) but the contents of the mod have changed.
...Klinn
Post Extras:
|
|
|
|
I remembered seeing something like this before, so I did some digging through my archived threads and turned up a post by Farren which I think explains what's going on in this instance. The original post has gone missing in one of Beths clean outs, so I'll re-post the whole thing;
=======================================
Author: Farren
Subject: Doubling [Re: Farren]
Post: #1637795 - 07/23/03 11:24 PM
Coda
I got around to reading the interesting thread you linked to and I think I understand the problem. I'm posting this in the hope you can help me brainstorm the algorithm for sorting it out.
On the face of it, your "remove differences" idea seems good. However, I've had situations where I don't discover the double until a significant number of real changes have occurred in my game. So this approach would still yield occasional difficulty.
I trying to figure out a more automatic approach. The problem seems to arise in two cases:
1) you load a plugin with a more recent timestamp than your save
2) You load an older plugin that references the same data as a newer plugin, _and your save game was saved with the newer plugin loaded_
Every object or character in the morrowind game world has two unique references:
1) The ID string, such as "ex_velothi_door" that uniquely identifies the _type_ of object
2) The cell reference, that uniquely identifies a particular _instance_ of that object
In Dave Humphrey's excellent ESM/ESP/ESS file format definition, he states that the FRMR subrecord found for each object/character in a cell is the unique _instance_ identifier:
Quote:
-------------------------------------------------------------------------------
FRMR = Object Index (starts at 1) (4 bytes, long): This is used to uniquely identify objects in the cell. For new files the index starts at 1 and is incremented for each new object added. For modified objects the index is kept the same.
-------------------------------------------------------------------------------
When the engine overloads object changes, it uses this reference to determine what to overload. It seems impossible using this method for the doubling to occur.
So I made a single change to an object and saved it as an esp. The FRMR records were _not_ the same. The first index is a smaller number and the second much larger. But in this case, the engine can't tell that the reference in the ESP is a modification of the ESM, since they don't have the same reference?!?
Further investigation. The FRMR record is made up of 4 bytes. In accordance with DHs file format, I was interpreting all four as a single LONG (Long Integer) type number. When I take the first byte and treat it as a number, and the next three and compound them into a seperate number, something interesting arises.
The three bytes of the changed reference are the same number, always, as the three bytes of the original reference. So there is a unique identifier number that tells you which object was changed by the ESP, with an extra "flag" byte of unknown purpose.
Unfortunately, I've just reconstructed my machine with new hardware and my programming tools and low level editor are giving errors, do I can't continue till I've fixed this, but here's my thesis:
When you manipulate or interact with something in Morrowind, a reference to the object is saved in your saved game file. The 3 byte number that uniquely identifies the object remains the same as the 3 byte number in the master and all overriding esp files, but the "flag" changes.
The flag can't be a time stamp because 1 byte isn't enough data to store that. However, in the ESS, the ESM/ESPs the save game is dependent on are listed at the beginning of the file, so the flag could be a reference in sequential order to the ESM/ESP that last overrode that object.
Now if you save with the newer mod installed, the newer mod would be the last in the list and the flag would be the maximum value. Installing an older mod, even though it's older, puts the older mod at the last in the list (before you save again the older mod won't be listed in your file, but it will be listed in memory).
If the engine loads the changes to objects like this
Highest flag value last (Flags in ESPs will be the same, but in save will be higher)
Highest date value last
There's probably some kind of conflict when the flags are different and the flag/date sequences are the reverse of each other, forcing the engine to duplicate and show a new object to resolve the conflict.
Hmm... unfortunately an upgrade made it difficult for me to check anything until my machine is sorted out but if something like this is the case the presence of a unique reference in all three files will make it possible for me to
1) Present an interface where you specify your save game
2) Select current mods (defaulting to those listed in the game)
3) Use the unique ref to determine the doubling objects and update the flag to prevent it
Anyway, I'll be able to see the relationships between flags, filestamps and references when I've got everything running again.
Post Extras:
|
|
Argent
|
|
Disciple
|
|
|
|
|
Reged: 02/05/03
|
|
Posts: 1847
|
|
Loc: Australia
|
|
|
Sheesh! Just when you think you've got a handle on the situation, MW throws in other enigma  . I don't have a MW installation on this machine, so I can't test it right at the moment.
Quote:
Has anyone looked at the contents of a save game file to see if a list of mods-in-use is in there?
Save files store 'mods-in-use' in the same way esp files store parent masters. When I get a chance, I'll look into this problem further.
--------------------
Immersive Architect: Make Morrowind your world builder.
Morrowind Tech
Post Extras:
|
|
DinkumThinkum
|
|
Diviner
|
|
|
|
|
Reged: 09/20/01
|
|
Posts: 4731
|
|
Loc: Luna Penal Colony
|
|
|
Quote:
Sheesh! Just when you think you've got a handle on the situation, MW throws in other enigma . I don't have a MW installation on this machine, so I can't test it right at the moment.
I've thought of a couple more experiments to try that might (or might not) come up with useful information about this and possibly make a start toward identifying that pattern Klinn mentioned.
Quote:
Quote:
Has anyone looked at the contents of a save game file to see if a list of mods-in-use is in there?
Save files store 'mods-in-use' in the same way esp files store parent masters. When I get a chance, I'll look into this problem further.
Thank you!
I have the technical background to poke around in file formats, dig through hex code (though I'm old-fashioned enough to be more at home in octal...), etc. But it's been a long time since I've done much of that, and almost none since I downgraded  to a Microsoft/Intel thingie.  So I'm rusty as heck, and not really enthusiastic about getting into the low level details of things any more. 
I will do a bit more experimenting shortly, but I'll probably have to leave it up to the gurus to figure out the details of just what the heck is going on. 
--------------------
How to avoid the most common problem encountered by new modders: Dirty Saves: Causes and Cures
Post Extras:
|
|
DinkumThinkum
|
|
Diviner
|
|
|
|
|
Reged: 09/20/01
|
|
Posts: 4731
|
|
Loc: Luna Penal Colony
|
|
|
Figured out part of what's going on here, and have a little better idea of the rest of it.
Loading the older mod was not overriding data in the saved game.
Reason: that chest wasn't in the save game. No data, so nothing to override.
Confirmed this by looking at the save game using the Details list in the Construction Set. The only reference listed for Foryn Gilnith's shack (in the saved game) was Foryn Gilnith, the NPC.
Next test:
1. I enabled just the Newer mod and loaded my clean save outside the Census Office. I went to Foryn's shack, and the chest was where the Newer mod placed it: next to the hammock.
2. This time, I put an item into the chest before saving the game. (In my previous test, I hadn't interacted with the chest at all).
3. Then I saved the game, and reloaded (that same save) with both the Older and Newer mods enabled.
4. Now the chest stayed where it was supposed to be; the Older mod didn't relocate it.
5. When I checked this save game with the Details list, the chest was now in the saved game for that cell.
----------------------------------------------------------------------------
So the Older mod only overrode the chest data stored in the Newer mod; it wouldn't override the chest data in the saved game.
This explains why my attempts to duplicate this by moving Arrille around failed: as an NPC, he was included in my saved game as soon as I entered his cell, even though I hadn't interacted with him.
But the chest was only stored after I interacted with it, so the game just had the changed data in the two mods to deal with.
I still have no idea why the game would pick the chest data from the Older mod. Only guess is that since the Newer mod was already listed as being a 'master' for the saved game and the Older mod wasn't, the game treated the Older mod as a newer addition and used that chest data instead of the data in the Newer mod.
Argent, maybe you (or anybody who's more familiar with the file formats than I am  ) can come up with an explanation of what's happening here, in case there's a usable work around to avoid this bug.
----------------------------------------------------------------------------
Changing the load order of mods is often suggested as a way of resolving conflicts, getting mods to work together better, etc.
Based on this, that may not be as reliable as we've all thought: sometimes an older mod can override a newer mod, even though the newer mod is lower on the load order list.
I still have some more things to check on this. Just wanted to post this while it was still fresh in my head.
----------------------------------------------------------------------------
----------------------------------------------------------------------------
Edit:
For completeness sake, I just did a quick test of both saves in Foryn's shack with neither of my test mods loaded.
1. When I loaded the save game in the shack, where my character had not interacted with the chest, the chest was back to its default position.
2. When I loaded the save game where my character had interacted with the chest (put an object in it), it was still in the changed position, even though the mod that moved it there was no longer loaded.
Which were the results I expected, but at this point I'm not sure which expectations are reliable any more.
--------------------
How to avoid the most common problem encountered by new modders: Dirty Saves: Causes and Cures
Edited by DinkumThinkum (07/24/04 10:59 PM)
Post Extras:
|
|
DinkumThinkum
|
|
Diviner
|
|
|
|
|
Reged: 09/20/01
|
|
Posts: 4731
|
|
Loc: Luna Penal Colony
|
|
|
Did some testing with 'references persist' checked on the container I was moving around, and that turned out to have an effect on which mod's data was used.
----------------------------------------------------------------------------
1. For the test, I created a tiny .esm file that set the 'references persist' flag for "chest_small_01_gold_10", which is the chest on Arrille's counter (Tradehouse, in Seyda Neen).
2. Then, with that .esm loaded, I created an Older and a Newer .esp, each of which moved the chest to a different position.
When I checked the Details list for the mods, the reference for the moved chest was now listed as a 'Reference'. (In the previous tests, without 'References persist' checked, it was listed as a 'Temporary Reference'.)
3. With only Newer loaded, I went to the Tradehouse and saved the game with the chest in the position set by the Newer' mod. (I just entered the cell; did not interact with the chest.)
(When I looked at the Details list for the saved game, the chest was still not listed. Even with 'references persist' checked, it still wasn't included in the save game.)
4. Then I exited the game, enabled both the Older and the Newer mod, and reloaded the save inside the Tradehouse.
5. This time, with 'references persists' checked and the container a reference (instead of a temporary reference) in the .esp files, the position in the Newer mod was used. In previous tests, the position in the Older mod overrode the position in the Newer mod when both were loaded.
----------------------------------------------------------------------------
I also tried the 'references persist' .esm mod, in the game, with Older and Newer .esp files that had been made without it. In those .esp files, the moved container was listed as a 'temporary reference''.
Even with the 'references persist' .esm file enabled, in the game I got the same results as I had without it: the chest position in the Older mod overrode the position in the Newer mod when both were loaded.
----------------------------------------------------------------------------
Apparently the controlling factor is whether or not the container is stored as a reference or a temporary reference in the .esp file; setting the 'references persist' flag doesn't seem to have any effect once the .esp is already created.
So, for my test conditions of saving with Newer mod loaded, then reloading that save with Newer and Older mod loaded:
If the item is stored in the .esp file as a reference, information in the Newer mod takes precedence.
If the item is stored in the .esp file as a temporary reference, the Older mod takes precedence.
This explains why, when I tested by putting Arrille in different positions, the Older .esp file didn't override the Newer one: as an NPC, Arrille is always listed as a Reference, not a Temporary Reference.
----------------------------------------------------------------------------
Note: I never did see the doubling effect that Soar182 reported, which is what started this testing session. However, with the weirdnesses I did find, doubling does seem like a possible result.
For the moment, I will leave it up to the gurus to figure out why it's working like this. I could probably come up with more experiments, but I can't think of any that would have much chance of telling us any more than what I've done so far.
Besides, my brain is fried...
--------------------
How to avoid the most common problem encountered by new modders: Dirty Saves: Causes and Cures
Post Extras:
|
|
Soar182
|
|
Curate
|
|
|
|
|
Reged: 05/30/03
|
|
Posts: 585
|
|
Loc: Minnesota
|
|
|
Here is what I have.
I have my house plug in.
There is a set of alchemy equipment and a ring on the table. When starting.
The ring is on me and the equipment is now in a crate.
That is the only plug in I am running with this saved game.
The plug in was cleaned with TESAME and checked for uncleans with TES Plugin Conflict Detector.
1. If I add a plug in older then my plug in. The ring and equipment reappear on the table.
2. If I load a newer plug in. There is no doubling
3. If I open and save the old plug in. In the CS.
Then clean it and check/remove uncleans. I can add to my saved game with no problems.
--------------------
Screen shots
Post Extras:
|
|
DinkumThinkum
|
|
Diviner
|
|
|
|
|
Reged: 09/20/01
|
|
Posts: 4731
|
|
Loc: Luna Penal Colony
|
|
|
Quote:
Here is what I have.
I have my house plug in.
There is a set of alchemy equipment and a ring on the table. When starting.
The ring is on me and the equipment is now in a crate.
That is the only plug in I am running with this saved game.
The plug in was cleaned with TESAME and checked for uncleans with TES Plugin Conflict Detector.
1. If I add a plug in older then my plug in. The ring and equipment reappear on the table.
2. If I load a newer plug in. There is no doubling
3. If I open and save the old plug in. In the CS.
Then clean it and check/remove uncleans. I can add to my saved game with no problems.
Thinks for the more detailed description of what you were doing to get doubling.
If I get some time, I'll do some more testing and see if I can duplicate the doubling. But I'm not sure when (or even if...) I'll get to it.
I don't know my way around the internal formats of the files used by Morrowind, and I think it will take that kind of knowledge to figure out what is going on here. Unfortunately, there only seem to be a handful of experts like that around, and most of them aren't posting much anymore.
Even though my tests didn't reproduce the doubling you saw, they definitely confirmed that there's some kind of bug or oddity in the way Morrowind handles data from .esp files with different dates. Data in an older .esp file should not override data in a more recent .esp, but in some cases it does.
--------------------
How to avoid the most common problem encountered by new modders: Dirty Saves: Causes and Cures
Post Extras:
|
|
Previous
Index
Next
Flat
Threaded
Possible new load order, dirty mod, dirty save, or something bug.
Edit
Reply
Quote
Re: Possible new load order, dirty mod, dirty save, or something bug.
[Re: 